Conducting Usable Privacy & Security Studies with Amazon’s Mechanical Turk

Being able to conduct human subjects experiments in a distributed method across the Internet is frequently desirable to support broad tests of usability. Until recently these experiments were commonly advertised in an ad-hoc fashion, using mailing lists, contest sites, and online bulletin boards. Recently Amazon’s Mechanical Turk, a service where users can complete short tasks and receive automatic payment, has become prominent in the HCI community. We describe three different usable privacy and security experiments that were conducted through Mechanical Turk, highlighting both reasons for using Amazon’s service as well as common pitfalls that we encountered.